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DISCLAIMER 

The  views  and  conclusions  contained  in  this  document  are  those  of  the 
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I. 


SUMMARY 


This  Interim  Progress  Report  summarize*  work  performed  under  Task  1,  ."Data 
Collection  and  Analysis  /'  of  the  contract  Statement  of  Work.  The  objective  of 
the  total  effort  is  to  explore  options  and  provide  recommendations  for  improving 
program  management  functions  related  to  technical.  r^Lsk  assessment  and  cost/ 
schedule  uncertainty  for  selected  DARPA  programs .  -%»e  effort  documented  in 
this  report  includes  an  analysis  of  historical  data  related  to  government  R&D 
program  cost  overruns  and  a  preliminary  assessment  of  the  implications  of  such 
analysis  for  management  reserves  for  related  high-risk  DARPA  programs .  This 
assessment  will  ultimately  lead  to  the  specification  of  a  process  to  guide  DARPA 
management  in  the  establishment  of  budgetary  reserves  to  account  for  technological 
risk  and  to  periodically  update  projections  of  selected  program  cost  growth 
based  on  experience.  Such  a  procedure  would  not  only  benefit  future , DARPA  program 
planning  and  control j  but  is  also  responsive  to  the  letter  and  spirit  of  recent 
recommendations  to  Improve  the  acquisition  process  with  the  Department  of  Defense 


(DOD) 


Cyrhe  key  to  the  work  performed  so  far  under  this  contract  is  the  establishment 
of  a  data  base  containitg  information  on  historical  cost  growth  of  major  government 
R&D  and  production  programs  during  the  period  FY  1977  through  FY  1S80.  These 
programs  include  many  diverse  systems  developed  by  the  Departments  of  Defense, 
Energy  and  the  National  Aeronautics  and  Space  Agency  (NASA),  including  DARPA.  The 
analysis  discussed  in  more  detail  later  in  this  report  Include  spectral  density 
analysis  of  cost  overrun  data  to  develop  logical  aggregations  within  the  overall 
data  base.  >Based  on  statistical  analysis  of  these  data  subsets,  trends  relating 
probabilistic  measures  of  cost  growth  and  uncertainty  were  developed ^  These 
trends  displayed  a  remarkable  consistency  for  all  data  sets  under  a  * ariety  of 
analytical  assumptions.  %  e  analysis  shows  that  based  on  historical  evidence, 


\  <  ■ 
\  / 


-  ^ 


<  U- 

"^cost  overruns  of  significant  size  (from  SO  to  100  percent)  are  not  uncommon.  In 
addition,  the  analysis  shows  that  there  is  a  predictable  relationship  between  the 
size  of  an  original  estimate  of  program  overrun  (i.e.,  management  reserve)  and  the 
confidence  level  that  eventual  cost  growth  will  remain  within  that  limit.  For 
the  initial  set  of  DARFA  programs  studied,  the  analysis  indicates  that,  at  the 
95  percent  confidence  limit,  DA&PA  cost  growth  factors  (i.e.,  the  ratio  of  actual 
program  costs  to  original  estimates)  lie  in  the  range  of  approximately  40  to  110 
percent..  The  analysis  also  permits  management  to  estimate,  in  the  aggregate,  what 
total  prAgram  management  reserves  should  be  as  a  function  of  confidence  limit. 

It  ia  noted  that  caution  should  be  used  in  the  application  of  this  preliminary 
analysis  t<^  current  or  future  DARFA  programs.  Limitations  imposed  by  the  size  and 
nature  of  the  data  base,  and  by  a  still  limited  understanding  of  the  causal  nature 
of  cost  growth  and  risk,  make  it  unadvisable  to  attempt  to  immediately  adopt  these 
results  as  a  tool  for  DARFA  management.  Nevertheless,  the  results  so  far  do 
suggest  that  further  study  may  lead  to  the  development  of  a  procedure  for  improving 
the  accuracy,  credibility,  and  defensibility  of  DARPA  cost  estimates  and  management 
reserves.  The  following  section  provides  a  more  detailed  discussion  of  these 
results. 


II.  TECHNICAL  RESULTS 


Cost  Growth  Risk 
Assessment  and  Prediction 
for  DARPA  Programs 


o  Examine  broadest  possible  high  technology  data  base  for 
actual  cost  growth  information 

o  Develop  rational  aggregations  of  the  data  base  and  search 
for  consistent  statistical  trends 

o  Relate  available  DARPA  program  data  to  broad-based  trends 

o  Establish  cost  growth  risk  relationship^)  from  broad  data 
base  that  are  both  statistically  and  rationally  applicable 
to  the  DARPA  experience  and  environment. 


DATA  BASE  CORRELATION 


o  Dimensionless /'cost  factovs"  used  for  graphical  correlation 
and  visualization. 

o  "Estimated  Cost  Growth  Factor"  is  the  ratio  of  the  estimated 
increased  total  cost  (baseline  cost  plus  cost  growth  estimate) 
to  baseline  (initial)  total  cost. 

o  ."Cost  Growth  Risk  Factor"  is  the  ratio  of  the  statistically 
established  probable  cost  growth  (to  some  level  of  probability) 
to  baseline  cost. 

o  The  45°  line  of  figure  1  represents  a  ."perfect"  cost  growth 
estimate,  the  dashed  .“Cost  Growth  Risk"  line  represents  the 
statistically  established  potential  deviation  from  the 
/'perfect  estimate". 

o  The  vertical  distance  from  the  /'Perfect  Estimate"  line  to  the 
/'Cost  Growth  Risk”  line  is  a  measure  of  the  additional  cost  risk 
Inherent  when  the  /'estimated  cost  growth"  is  anticipated. 

o  The  slope  of  the  /'Cost  Growth  Risk"  line  will  be  determined  by 
statistical  (.nalysis  of  actual  DOD-DOE  and  NASA  cost  growth 
experience  for  FY  '77,  '78,  *79  and  '80,  correlated  with  all 
similar  available  DARFA  data. 


STATISTICAL  PkOCESS 


o  The  DOD-DOE-NASA  data  base  of  actual  :ost  overrun  experience 
was  aggregated  Into  statistical  groupings  by  both  spectral 
density  analysis  and  fiscal  year  groups  of  cost  growth 
factors  (actual  cost  over  estimated  cost) . 

o  The  mean  value  and  standard  deviation  were  established  for 
each  group  or  aggregation  of  cost  growth  factors  using  the 
statistical  standard  distribution. 

o  The  cost  growth  risk  line  was  established  for  these  data  at 
the  mean  of  the  two-standard-deviation  (2-sigma)  values  for 
the  established  groupings. 

o  The  process  was  repeated  for  DARPA  data,  resulting  in  figures 
2  and  3  wherein  each  square  is  represented  as  i"2-sigms”  long 
to  a  side,  wih  the  mean  cost  growth  factor  value  at  the  lower 
7 eft  hand  corner. 

o  Flguru  2  presents  DARPA  data  grouped  by  spectral  density 
analysis  and  figure  3,  grouped  by  fiscal  year.  The  dashed 
line  in  each  case  is  the  DOD-DOE-NASA  /’Cost  Growth  Risk” 
boundary . 
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DARPA  PROGRAM  COST  GROWTH  RISK 
(Spectral  Density  Grouping) 


Coat  Growth  Risk  Factor 
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DARPA  PROGRAM  COST  GROWTH  RISK  by  YEAR 
(Fiscal  Year  Grouping) 


Estimated  Cost  Growth  Factor 


Figure  3 


CONFIDENCE  LEVELS 


AND  MANAGEMENT  RESERVES 

Figure  4  was  developed  by  taking  the  mean  DARPA  cost  growth 
risk  slope  from  figures  2  and  3,  reapplying  the  statistical 
standard  distribution,  and  establishing  /‘bench  mark"  levels 
of  confidence  of  cost  overrun  avoidance. 

Tables  1  and  2  illustrate-  a  preliminary  and  basic  application 

of  these  confidence  levels  in  determining  the  amount  of 

management  reserve  which  must  be  set  aside  to  reach  the 

various  levels  of  confidence  of  cost  overrun  avoidance  for 

* 

specific  DARPA  programs. 

Tables  1  and  2  are  meant  to  be  merely  demonstrative,  in  a 
simplistic  manner,  of  one  mode  of  application  of  the  actual 
experience  data  base.  Further  development  will  result  in 
the  refinement  of  this  analytical  tool  and  its  synthesis 
with  other  methodologies  into  an  integrated  cost  and  cost-* 
risk  management  system. 


COST  GROWTH  RISK  RESERVE  FACTOR 
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